Support managed blue/green Flex Node agent upgrades by bcho · Pull Request #266 · Azure/AKSFlexNode

hbc (bcho) · 2026-08-06T21:19:42Z

Summary

handle Unbounded AgentUpgrade MachineOperations in the Flex daemon
add a host-driven agent-upgrade [--preflight] command for separately delivered candidates
migrate legacy direct-file installs into an idempotent blue/green binary layout
reuse the merged public pkg/agent/agentbinary APIs for activation planning, candidate snapshots, locking, verification, switching, health checks, and rollback
accept HTTP(S) release archives with optional compressed-archive SHA-256 while enforcing exact architecture members, size bounds, and candidate version verification
synchronize the active nspawn exec-credential binary with the selected host binary
persist upgrade state across daemon restarts and recover verified last-good host/nspawn binaries through systemd
coordinate direct and MachineOperation activation with a shared host lock and durable pending signal
redact credential-bearing URL query strings from logs and operation status
add daemon MachineOperation RBAC, operations documentation, and focused real-cluster E2E coverage

Architecture

Flex supplies its own unexported daemon-service adapter for aks-flex-node-agent.service. It reuses the existing Flex lifecycle assets, utilexec/utilio, persisted active-nspawn state, and recovery behavior rather than importing Unbounded command internals or copying Unbounded-specific unit assumptions.

The dependency is pinned to merged Azure/unbounded#592 commit b05575d6910044834722daaccffe6d4304e1f70a.

install.sh remains unchanged and owns only normal installation. E2E/provisioning rejoin logic detects an existing managed layout and invokes the separately uploaded candidate's Go agent-upgrade command before bootstrap.

Upgrade lifecycle

Validate parameters and acquire the shared host activation lock.
Persist the durable pending signal before transitioning the MachineOperation to InProgress.
Download and verify the archive and candidate in the inactive slot.
Atomically switch host links and synchronize the active nspawn rootfs.
Schedule restart through a separate transient systemd unit.
On startup, gate the serialized repave/MachineOperation delegates until cache readiness, a bounded stability interval, and durable result publication or recovery complete.
Persist the committed-switch phase and roll back only when activation committed or the candidate is currently active, using a bounded context independent of daemon shutdown cancellation.
Preserve recovery state until a last-good process is confirmed running after failure.

Direct host activation privately snapshots the staged candidate, performs the same managed switch, and leaves an already inactive service stopped for reset/rejoin provisioning. MachineOperation and direct activation cannot overlap: the process lock protects staging/switching, and the durable signal protects the post-restart handoff window.

Security behavior

HTTP and an omitted digest intentionally follow the merged Unbounded MachineOperation contract for control planes that trust the source and transport path; production guidance recommends HTTPS plus SHA-256.
Archives are bounded and must contain exactly one architecture-specific member (aks-flex-node-linux-amd64 or aks-flex-node-linux-arm64).
Candidates must pass the version command before activation.
Production upgrade paths and signals remain root-owned.
Query strings and fragments are omitted from logs and operation status.

Testing

make verify && make check && make build-all
shell syntax validation for E2E and recovery scripts
focused unit coverage for migration recovery, optional digests, lock contention, status/signal ordering, conditional rollback, inactive-service activation, and service assets
real AKS/Flex Node E2E coverage for:
- digest-verified MachineOperation success
- forced daemon failure and automatic last-good rollback
- retry without the optional digest over VM-local loopback HTTP
- direct CLI preflight and blue/green activation
- host/nspawn digest equality
- kubelet restart, Lease renewal, and Node readiness
- unjoin/rejoin using separately staged candidate activation
- workload smoke tests and cleanup

Run the focused scenario with:

./hack/e2e/run.sh agent-upgrade

Fixes #265

hbc (bcho) · 2026-08-06T21:36:11Z

Temporarily reopening to retrigger the repository's native PR checks; keeping this PR in draft.

hbc (bcho) · 2026-08-06T21:56:37Z

Dependency update: the secure HTTPS/SHA-256/archive/blue-green installer was moved to Azure/unbounded#585. This PR now consumes Unbounded commit 5e0aab6f, removing about 430 lines of duplicated FlexNode implementation and tests.

hbc (bcho) · 2026-08-06T22:08:33Z

Self-review updates pushed in 3448ea9: updated to Unbounded 8173c7ee, removed goal-state type coupling, and now converges the blue/green layout plus recovery service/script and OnFailure unit on every daemon startup. Added an in-place legacy service/binary migration test.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

hbc (bcho) · 2026-08-06T22:23:22Z

Final focused real-cluster E2E passed on the updated implementation: successful MachineOperation upgrade, host/nspawn digest equality, Node readiness, kubelet Lease renewal, forced candidate failure with automatic last-good rollback, sensitive URL status check, retry into the inactive slot, and workload smoke test.

hbc (bcho) · 2026-08-06T22:37:43Z

Updated the dependency to Unbounded 72a21798, where Unbounded's own managed AgentUpgrade path now uses the same secure installer.

hbc (bcho) · 2026-08-06T22:43:47Z

Updated to Unbounded 75333be3, which removes the duplicate internal staging implementation/tests after migrating Unbounded production to the shared secure installer.

hbc (bcho) · 2026-08-07T00:03:23Z

Updated to Unbounded d68bbe5c, which fixes the upstream AgentUpgrade E2E for the new HTTPS + sha256 contract.

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

docs/usages/operations.md:80

This preflight invocation also needs root privileges. The managed links live under /usr/local/lib/aks-flex-node, whose directory is created with mode 0750; a normal user cannot resolve the current/last-good links, so this documented command fails before producing a plan. The E2E correctly invokes preflight with sudo.

/var/tmp/aks-flex-node-candidate agent-upgrade --preflight

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated no new comments.

Suppressed comments (2)

pkg/daemon/agent_upgrade.go:125

RecoveryRequired is set even when no switch occurred. If staging fails before activation and Abort then fails while inspecting or clearing the signal, beginAgentUpgradeRecovery records this flag; startup recovery consequently restores LastGoodPath, potentially downgrading a still-valid current binary after an earlier successful upgrade. Derive this flag from SwitchCommitted/whether the candidate is active, and track a restart needed only for result publication separately.

	signal.RecoveryRequired = true

pkg/daemon/agent_upgrade.go:334

Mandatory: clearing the signal here is not crash-safe with the callers. The stage/restart/wait failure branches call Abort and only afterward persist the terminal failure; because the operation is already InProgress, a crash after this removal leaves no recovery signal, and the shared reconciler does not enqueue InProgress operations. Split rollback from signal deletion and retain the signal until terminal status is durable, or let startup publish the failure.

	return e.signals.clear()

# Conflicts: # pkg/daemon/daemon.go

# Conflicts: # cmd/aks-flex-node/main.go # go.mod # go.sum # hack/e2e/README.md # hack/e2e/run.sh

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated no new comments.

Suppressed comments (1)

pkg/daemon/agent_upgrade.go:395

recordFailure unconditionally sets RecoveryRequired, so a systemd failure before the candidate is switched still forces rollbackAgentUpgradeFiles to restore last-good. After a prior successful upgrade, last-good is the older slot; an unrelated crash during RecordPending, status update, or download therefore downgrades a valid current binary even though the candidate was never active. Determine recovery necessity from SwitchCommitted/whether CurrentPath resolves to CandidatePath, while preserving an already-recorded recovery requirement, before invoking rollback.

	if err := signals.recordFailure(message); err != nil {
		return err
	}
	signal, err := signals.read()
	if err != nil {

Copilot

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated no new comments.

hbc (bcho) · 2026-08-12T18:23:58Z

manual validate in test env, will merge it and validate in a separate PR

feat: add managed agent binary upgrades

929fa13

hbc (bcho) closed this Aug 6, 2026

hbc (bcho) reopened this Aug 6, 2026

hbc (bcho) marked this pull request as ready for review August 6, 2026 21:36

Copilot AI lite review requested due to automatic review settings August 6, 2026 21:36

hbc (bcho) closed this Aug 6, 2026

hbc (bcho) reopened this Aug 6, 2026

hbc (bcho) marked this pull request as draft August 6, 2026 21:36

hbc (bcho) mentioned this pull request Aug 6, 2026

Agent: export secure binary upgrade installer Azure/unbounded#585

Merged

refactor: reuse unbounded agent upgrade installer

5cba818

fix: converge upgrade recovery on daemon startup

3448ea9

Copilot AI reviewed Aug 6, 2026

chore: update shared agent upgrade dependency

9fe1ad1

chore: update shared upgrade implementation

f437ac9

chore: update reviewed upgrade dependency

0f3c109

hbc (bcho) had a problem deploying to e2e-testing August 6, 2026 23:09 — with GitHub Actions Failure

Copilot started reviewing on behalf of hbc (bcho) August 6, 2026 23:32 View session

chore: update agent upgrade dependency

85da2b3

hbc (bcho) had a problem deploying to e2e-testing August 7, 2026 00:03 — with GitHub Actions Failure

chore: update agent upgrade dependency

604ef21

hbc (bcho) had a problem deploying to e2e-testing August 7, 2026 00:13 — with GitHub Actions Failure

chore: update agent upgrade dependency

2155e9e

hbc (bcho) had a problem deploying to e2e-testing August 7, 2026 00:21 — with GitHub Actions Failure

Copilot AI reviewed Aug 11, 2026

View reviewed changes

Comment thread pkg/daemon/machineoperation_reconciler.go Outdated

Comment thread pkg/daemon/daemon.go Outdated

Comment thread pkg/daemon/host_agent_activation.go Outdated

fix: retry unresolved upgrade handoffs

e110e34

hbc (bcho) temporarily deployed to e2e-testing August 11, 2026 05:17 — with GitHub Actions Inactive

hbc (bcho) requested a balanced review from Copilot August 11, 2026 05:19

Copilot started reviewing on behalf of hbc (bcho) August 11, 2026 05:19 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

Comment thread pkg/daemon/machineoperation_reconciler.go

fix: hold reconciliation through daemon handoff

11e42ca

hbc (bcho) temporarily deployed to e2e-testing August 11, 2026 05:42 — with GitHub Actions Inactive

fix: preserve recovery handoff serialization

176d945

hbc (bcho) temporarily deployed to e2e-testing August 11, 2026 05:44 — with GitHub Actions Inactive

test: cover successful upgrade publication

09befda

hbc (bcho) temporarily deployed to e2e-testing August 11, 2026 05:48 — with GitHub Actions Inactive

hbc (bcho) requested a balanced review from Copilot August 11, 2026 05:56

Copilot started reviewing on behalf of hbc (bcho) August 11, 2026 05:57 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

Comment thread pkg/daemon/agent_upgrade.go

Comment thread pkg/daemon/host_agent_activation.go

hbc (bcho) commented Aug 11, 2026

View reviewed changes

hbc (bcho) requested a balanced review from Copilot August 11, 2026 06:06

Copilot started reviewing on behalf of hbc (bcho) August 11, 2026 06:07 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

Comment thread pkg/daemon/assets/aks-flex-node-recovery.sh

hbc (bcho) requested a balanced review from Copilot August 11, 2026 06:14

Copilot started reviewing on behalf of hbc (bcho) August 11, 2026 06:15 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

fix: let restarted daemon own terminal status

f977bbd

hbc (bcho) temporarily deployed to e2e-testing August 12, 2026 03:45 — with GitHub Actions Inactive

hbc (bcho) added 2 commits August 12, 2026 17:05

Merge remote-tracking branch 'origin/main' into hbc/daemon-upgrade

11087d6

# Conflicts: # pkg/daemon/daemon.go

Merge remote-tracking branch 'origin/main' into hbc/daemon-upgrade

9bfa30c

# Conflicts: # cmd/aks-flex-node/main.go # go.mod # go.sum # hack/e2e/README.md # hack/e2e/run.sh

Copilot AI reviewed Aug 12, 2026

View reviewed changes

Thalia Wang (wenxuan0923) approved these changes Aug 12, 2026

View reviewed changes

Uh oh!

Conversation

hbc (bcho) commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Architecture

Upgrade lifecycle

Security behavior

Testing

Uh oh!

hbc (bcho) commented Aug 6, 2026

Uh oh!

hbc (bcho) commented Aug 6, 2026

Uh oh!

hbc (bcho) commented Aug 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

hbc (bcho) commented Aug 6, 2026

Uh oh!

hbc (bcho) commented Aug 6, 2026

Uh oh!

hbc (bcho) commented Aug 6, 2026

Uh oh!

hbc (bcho) commented Aug 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

hbc (bcho) commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

hbc (bcho) commented Aug 6, 2026 •

edited

Loading